RDMA Reads: To Use or Not to Use?

Fast networks with RDMA are becoming common in data centers and, together with large amounts of main memory in today’s machines, they allow unprecedented performance for traditional data center applications such as key-value stores, graph stores, and relational databases. Several systems that use RDMA have appeared recently in the literature, all showing orders of magnitude improvements in latency and throughput compared to TCP. These systems differ in several aspects. Of particular interest is their choice to use or not to use RDMA reads. RDMA reads are appealing as they perform better than RPCs in many cases, but RPCs are more flexible as the CPU can perform additional operations before replying to the client. For example, systems that use RPCs often use indirection per object which makes it easier to de-fragment memory, evict cold objects to a storage tier, support variable sized objects, and perform similar tasks. Indirection per object can be implemented in systems that use RDMA too, but it doubles the cost of object reads, so it is avoided. The designers of RDMA-based systems are left in a dilemma: should they use RDMA reads or not? We help answer this question by examining the trade-off between performance and flexibility that the choice implies.